1. Field of the Invention
The invention comprises a method to increase the depth of focus (or field) for photography, and means for the method. The method constitutes an improvement of the depth of focus in use of traditional T / S xe2x80x94technology, xe2x80x9cTilt and Shiftxe2x80x9d, and an increased possibility for improved depth of field with other cameras. The method is mainly aimed for digital cameras, where the image is created from a sensor in the camera, instead of on a photographic film. By use of present scanning technology however, it is easy to transfer a film photo to an electronic image. The method can be used also for those cases. In the following the method is mainly presented for the cases, when the picture is created electronically directly in the camera.
2. Description of the Related Art
In T / S cameras one or both of the lens and the xe2x80x9cimage planexe2x80x9d are movable. The image plane is that plane, where the film or the sensor is positioned. A plane in the scene is sharply focused in a xe2x80x9cfocused planexe2x80x9d, which might be localized in front of, back of or cutting the image plane. Proportional to the distance between the image plane and the focused plane, the blurring grows larger in the image. In T / S cameras the photographer often views the image on a focusing screen positioned in the image plane. He controls the camera settings for a focused image on the focusing screen. Then he places the film or sensor in this position and shoots.
Photographers use many different ways of working. The profession is often creative, sometimes artistic. In practice the photographer puts the camera in position directed to the subject, arranges the image plane in the desired perspective, usually vertical, as we are used to view the world from a vertical perspective. Tilting an ordinary camera upwards, e g for placing a church tower inside the seeker, the camera will produce an image of the tower with a tilting perspective, showing a tower tilting or falling in. Parallel vertical lines are no more parallel in the image.
Then the photographer focuses and positions the subject for the image. Here difficulties might arise, which the invention can solve. The focal aperture is not only used for control of the light strength, but also for increasing the depth of field, e g at product photography. Increased exposure can compensate smaller apertures. However long time of exposure implies risks for mechanical disturbances blurring the sharpness.
T / S cameras are used for product photography, and when there is a desire to increase the depth of field from what an ordinary camera can offer. A close picture on a product with large extension in distance, might be given focus both on the front and back parts, by tilting the focal plane, cutting the front as well as the back part of the product image. The tilting might be arranged in two ways. One by positioning the film or sensor in the tilting image plane. The other by tilting the lens instead, turning the focal plane into the desired angle. Usually a vertical image plane is desired, obtaining a vertical perspective, and then there is just the method of tilting the lens, until the focal plane is positioned vertically in the desired image plane.
As subjects generally are not extended just in a plane of two dimensions, but are three-dimensional bodies with varying extensions, also focal aperture reductions are used to decrease the angle of refraction from the lens. Thus the image plane can be farther from the real focus, without too large blurring errors. However the subject or the light conditions might reduce the possibilities.
In the field of television the image experienced by the human beings has always been created by the use of many xe2x80x9csubimagesxe2x80x9d integrated together. By showing several frames (50 to 60) per second, the subimages are integrated in time, in order to show smooth movements, increase the resolution and the signal to noise ratio. A study of only one subimage reveals a poor image quality. An ordinary still camera, in contrast gives a single shot image of excellent quality, including much higher resolution and very low noise content.
In the field of research for high definition television, there has been an increased awareness of second order effects of the increase of image resolution. The one that will be discussed here is the aspect of depth of focus.
For a television image with low resolution, there is a large span of subject distances that can be xe2x80x9cfocusedxe2x80x9d to a single image plane, because the blurring of the optics is anyhow less than the resolution of the sensor detectors. However increasing the sensor pixel resolution e. g. with a factor 4 means a decrease of the depth of focus with a factor 2.
For television it is an obvious method to increase the depth of focus, by using the natural stream of new video images integrating those together, while changing the focus setting.
There are several similar patents in the field of television using this method of increasing the depth of focus. Most of those patents use transfer of the ordinary X,Y -based image to a space frequency based image description. They are using power comparisons in the space frequency or frequency filtered band pass domain, between subimages to define which subimage is in focus. They are differing in the method details of performing this object.
One method according to Mimura et. al. (U.S. pat. No. 5, 282,045) uses changing of the focus distance between frames, and then utilizing the integration of subimages in the mind of human viewers. The result is said to increase the experienced depth of focus in the mind of humans.
The method also includes a xe2x80x9ccontour correctorxe2x80x9d, which function is not described, but is implicitly an ordinary artificial xe2x80x9ccontour sharpenerxe2x80x9d, said to add higher frequencies to the contour, which after integration in the mind of humans, is experienced as a higher resolution image.
There are many different problems involved in those television applications, e g.
There are time restrictions. Only a short time is available for calculations and mechanical focusing, due to the high image frequency.
By the selection process of focused frames, the number of integrations decreases and the noise increases, decreasing the image quality.
In television there are generally movements, by subjects in the scene, or the camera panning or zooming the scene. In fact the movements are inherent in the television approach, and necessary for the human expectations on the television media. Then there are small possibilities for methods utilizing exact matching of successive subimages with different focus, as the subimages contains other changes due to movements. In the patent of Mimura et al, the depth of focus function is simply put off, when there are movements.
On the other hand the television is in many ways xe2x80x9cforgivingxe2x80x9d image deficiencies. A poor image is immediately followed by new ones in the continuous video-stream of images. The short time involved means that many imperfections pass the viewer unnoticed. Then a process of selecting focused image parts and combining those together can be allowed to involve some inaccuracies and non-perfect matches, without disturbing the viewer""s total integrated experience of the television video-image quality. This is opposite to the application of a still camera. The still photo is viewed for much longer time periods, and all imperfections and small details are also time integrated to noticeable and disturbing image quality problems. In the professional business, e. g. in case of a new company product photo, there is no forgiveness to be expected for even small image quality deficiencies.
The present invention concerns a method to achieve sharpness and depth of field for a still photo by use of multi-shots, where different camera settings can be used including aperture reduction technique. The method utilizes possibilities for digital image transfer after the shot, and the involved processes might be performed using automatic methods to various extents. An operator might also interact on computer screens, manually improve and control the result of the processes.
The camera sensor is turned into a suitable focal plane and the photo is shot.
Then the perspective can be turned into a desired position by image transformation. One way to do this is sensing the sensor turning in relation to the desired perspective and feed data to a calculation unit, which calculates how much the image should be turned and makes the transformations. E. g. if the image is tilted an angle u related to the desired vertical plane, then the image plane can be turned around its front edge by u.
Then image points at the distance a from the axis would be scaled to the distance a*cos(u) from the axis. Those points are also moved closer to the lens, a distance a*sin(u), which means the same reduction of scale as moving the image the corresponding distance closer to the lens, i. e. a distance b to the lens related to (a*sin(u)+b). This is easily illustrated by drawing radiation lines through the optical center to respective image point. Thus turning the image around the axis, the points on the image will be projected through simple geometrical rules to the new image plane.
The choice of turning axis can be transferred as turning around another axis plus a transversal movement. Transversal movements of image elements, changing the distance from the lens, result in a proportional change of scale of the image element. Thus ordinary focusing gives rise to size changes. Knowledge about the relative image distance from the lens (the optical center) would be used for the size scale.
When there are defocused image parts, also the defocused area will be enlarged proportional to a general image enlargement. A defocused edge would be wider, if special actions are not performed. Such actions are included in the invention and will be presented further in the section about enlargement.
Projections of images might be performed as second steps, in turning to the right image perspective. The operator might fine tune or control the total process, by introducing the desired axis and turning. He can also define a certain speed of turn and study the image turning until it reaches the desired position, when he can stop the turning or interactively change the turn directions, fine tuning the right position.
The calculations can be performed using a basic image, which has been mapped to an increased number of image pixels. Then unnecessary losses of resolution can be avoided through the calculations and image processing.
The control setting of desired focus planes at photography can be simplified and automated by the photographer setting the focus successively on those points, through which he wants the plane of focus. With known coordinates and focus settings the focus plane can be calculated and any turning of lens or sensor also. Simple calculation programs, including the camera optics, can be stored in the pocket calculator, which can present the result as scale partitions for a turning control. With a PC or other computers the operator can work with the image on the computer screen. Programs in the computer can control automatic focus measurements and calculation of focal plane and finally control turning and movements of the lens (sensor) to the right position. Thus the process can be performed as a production process, which is often wanted.
Creation of depth of field by use of several xe2x80x9cstillsxe2x80x9d.
At product photography the scene is often static, i. e. several photos can be shot reproducible. A product with relatively large extension in distance, can be photographed from the same position with focus set for different parts for the different photos. On the different photos, different parts are respective focused and defocused. They are also of different size.
Simple addition of image elements.
A sharp image of the product can be created by adding together focused different image elements from respective image. The process is made possible by first changing the image scale to the same size, according to sections above. In many situations the operator might do the element selection on a computer screen and determine from which image respective element shall be transferred to the final image.
The process can also be automatic according to the invented methods. The focus for an image sub-area is compared between images and the respective focused image element is selected. The division borders between neighboring image elements are chosen, e g where the defocusing is about the same from the different images. Any small deficiencies in the division interfaces might e g be handled by averaging that area between the respective images. Simple product shapes such as a cereal box, might be photographed from a perspective, which gives a corner the shortest distance to the camera and other parts successively farther distant. A series of photos with successively changed focus can produce the said focused image elements for image element additions.
The cereal box photographed inclined from a high front position, shows three surfaces; a long, a short and an upper side. By tilting the sensor or lens, a focused plane might be positioned along each of the three apparent surfaces. The three images are transformed by calculations to the selected basic image. If the sensor is turned, then the transformation also includes perspective returning.
After the calculations the product image can be combined from the three sharp image elements. In practice small deviations might have arisen in positions and sizes between the three different photos. Disturbances might have shaken the positions and controls. So sharp borders and edges are identified for the different images and common comers and boundaries are matched together, through fine tuned corrections of position and scale for the different images.
The basic image might e. g. be an image with the right perspective e. g. a vertical sensor and a lens setting giving a focus plane for a product""s front, or some other important part of the product. This basic xe2x80x9cimagexe2x80x9d can be defined by sensor and lens settings and be the defined xe2x80x9cbasic imagexe2x80x9d without even being photographed, also if a photo would have simplified the matching of the different photos to the basic image.
When additional pictures are desired for the integration of a sharp cereal box, then an extra focal plane might be positioned along the outmost left and right edges and through the diagonal of the upper surface, for helping adjustments of the said surfaces. When the table surface is of interest, on which the box is standing, then a focal plane might be positioned along this surface for an additional photo.
The integration of the final image might in this example be relatively simple, as each sharp element is easily identified and the defocused areas are substituted by sharp areas from other images.
Recalculations of perspective and scale of size is done to the defined basic image format, with the help of known settings for the sensor and lens. Analyses afterwards without knowledge of the settings, to find out the image position relative to another image position, can be performed, but it might need some more processing.
Corrections at the integration of respective sharp image elements are performed using common image parts e. g. edges and comers. Often there are image and text details, which simplify the match. If the adjustments of an image wouldn""t be good enough, e. g. dependent on a difficult perspective change, then local adjustments can be done for the different image elements. The sharp details are positioned in agreement with information from other images, and where distances in between are needed to be increased or decreased in a noncoherent way, the local area selected therefore is preferable an area with slow light and color variations.
Complex products.
Products might be more or less complicated in structure. There might be arms, poles, holes etc. Those can be identified by operators as special areas, xe2x80x9cmountains and valleysxe2x80x9d, where a small area might contain large distance variations, and whereby several photos might be needed to cover the dynamics. Areas with mountains and valleys can also be identified automatically according to the invention. They are areas, which are characterized in causing strong defocusing in consecutive images, although the neighboring image has sharp elements and though the transversal distance is close between the sharp parts of respective image. The operator can be alarmed about such and other problem areas by blinking on the screen, possibly at the position and with side comments. Often the operator might find focal planes, which can be positioned along steep mountains. And with additional photos, better integrated sharpness can be obtained for such areas.
If the product has an arm or a pole, then one or more images might be shot with focal planes through the arm, such that this subject is shown as a sharp object. In a corresponding way as for the cereal box, those images are transformed to a basic image for integration with the other images on the product.